Rule-based presentation of related content items

ABSTRACT

Features are disclosed for presenting multiple content items based on one or more rules defining how the multiple content items are to be presented. One content item may be a primary content item and any number of additional content items may be related to or supplement the primary content item in some way. Rules may define which related content items may be presented and, in the case of multimedia content (e.g., content with both audio and video elements), which elements of the primary content may be overridden by or mixed with related content, and which may not. References to related content may be embedded in the primary content item, or references may be accessed at a content management system or service.

BACKGROUND

Electronic devices may be used to consume content, such as audio books,television shows, movies, and music. In a common application, a user mayview video content or listen to audio content on a home entertainmentsystem. Home entertainment systems can include any number ofpurpose-built media devices, such as televisions, stereos, video gamesystems, and desktop computers that provide rich content consumptionexperiences. Users may also use personal computing devices to consumecontent. For example, a user may use a smart phone, tablet computer,portable media player or laptop computer to consume content. Suchdevices may include both audio and video output components, or they mayinclude only an audio output component.

Television shows, movies, audio books and the like can include topicsand items that may be associated with supplementary or otherwise relatedcontent. The items of related content often contain useful orinteresting information, such as additional background or explanatorytext regarding an event, location or person referenced in the primarycontent, commentary regarding the primary content in general or aclosely related topic, and the like. For example, a television show mayreference a historical event, or it may cover only part of a subject.Viewers may be invited to consume additional content related to, e.g.,the historical event or other areas of the subject. In some cases,viewers may be provided with a network address of a content server atwhich they can access the content, or viewers may be shown a commercialor other promotion for another television show with related content.

BRIEF DESCRIPTION OF DRAWINGS

Embodiments of various inventive features will now be described withreference to the following drawings. Throughout the drawings, referencenumbers may be re-used to indicate correspondence between referencedelements. The drawings are provided to illustrate example embodimentsdescribed herein and are not intended to limit the scope of thedisclosure.

FIG. 1A is a block diagram of illustrative interactions between a mediadevice, a content management system and a content server, as might occurduring presentation of a content item with related content items.

FIG. 1B is a block diagram of illustrative interactions between a mediadevice and multiple content servers, as might occur during presentationof a content item with related content items.

FIG. 2 is a block diagram of an illustrative media device, contentmanagement system, and content servers showing various modules and datastores of the media device and the content management system.

FIG. 3 is a flow diagram of an illustrative process for presenting acontent item and related content items according to rules forpresentation of related content.

FIG. 4A is a timeline of presentation of a primary content item andseveral related content items which may be presented in full or in partat various points during presentation of the primary content item.

FIG. 4B is a timeline of presentation of combined content comprisingportions of a primary content item and portions of related contentitems.

FIGS. 5A-5D are pictorial diagrams of illustrative user interfacespresenting a primary content item and several related content items,such as those on the timeline of FIGS. 4A-4B.

FIGS. 6A-6B are pictorial diagrams of illustrative user interfaces forconfiguring rules regarding the presentation of related content duringpresentation of a primary content item.

DETAILED DESCRIPTION Introduction

Generally described, the present disclosure relates to managingpresentation of multiple related content items, in some casessimultaneously or substantially simultaneously. Conventionally, a usermay use a media device such as a television or computing device topresent a content item. The content may include, but is not limited to,audio books, music, television shows, movies, games or some otherelectronic content. The content may be associated with supplemental orother related content, such as background content regarding a characteror event. In order for the user to access such related content, the usermay have to perform a manual process of stopping or pausing presentationof the primary content item in order to access the supplemental content.In some cases, users can access supplemental or related contentsimultaneously with consumption of the primary content item, such as byusing a different device (e.g., a computing device) to access therelated content while the primary content is presented on thetelevision.

Aspects of this disclosure relate to identifying and automaticallypresenting supplemental or related content during presentation of aprimary content item. Presentation rules, also referred to herein simplyas rules, may be defined which specify which types of related contentmay be presented, under what circumstances the related content is to beautomatically presented or suppressed, which media elements or otherportions of a primary content item (e.g., audio or video tracks of amovie) are to be overridden by or mixed with related content, and thelike. In some embodiments, a primary content item may contain embeddedreferences to supplemental content that is related to the primarycontent item as a whole or to a specific portion of the primary contentitem, such as a person, place, thing, event or period of time. When suchan embedded reference is encountered during presentation of the primarycontent item, the referenced supplemental content item can be accessedand automatically presented according to one or more rules. In somecases, the rules may indicate that the related content item (or someportion thereof) is to override the primary content item (or someportion thereof). For example, when the primary content item is atelevised sporting event and the related content item is a local radiobroadcast of the sporting event, a rule may specify that the mediadevice is to automatically present the audio element or other audioportion of the local radio broadcast instead of the audio element orother audio portion of the primary content.

The example rules described herein are illustrative only, and notintended to be limiting. In some embodiments, rules may be defined withrespect to various aspects or elements of content items (e.g., somerules apply to audio elements or other audio portions and some rulesapply to video elements or other video portions), genres of content(e.g., some rules apply to televised sporting events while other rulesapply to audio books), and the like. Generally described, a rule canspecify which content type and/or media element may override anothercontent type and/or media element, and under what circumstances suchoverrides may be implemented. Rules may also specify which content typeand/or media elements may be overridden, and under what circumstances.In addition, rules may specify that certain content is not to beoverridden, but rather is to be paused or should continue to bepresented simultaneously with other content using audio mixing or splitscreen video display techniques. Further, rules may specify whenoverrides may be applied automatically, and when users may be notifiedof the presence of a related content item and may therefore make anaffirmative decision regarding the presentation of each content item.

Additional aspects of the disclosure relate to obtaining references torelated content from a network-accessible content management system.Rather than references to related content being embedded in a primarycontent item, the references may be maintained by and obtained from acontent management system. The references may be obtained before orduring playback of the primary content item. For example, when a userinitiates presentation of a content item (e.g., a conventional DVDwithout embedded references to related content) on a media device, themedia device may contact the content management system to obtainreferences to related content and data regarding portions of the primarycontent item to which the related content applies. As another example, auser may initiate presentation of a live content item (e.g., a livebroadcast of a sporting event) in which some or all related contentreferences may be unknown at the time presentation is initiated. In suchcases, the media device can obtain related content references from thecontent management system in substantially real-time, or periodicallyduring presentation of the primary content.

Further aspects of the disclosure relate to user interaction with thecontent items and user configuration of presentation rules. For example,the user may access an interface to define presentation rules, setdefaults, and the like. In addition, rather than automaticallyoverriding portions of the primary content item with related content, auser may be prompted as to whether the primary content is to beoverridden or paused, or whether the related content is to be ignored.For example, when a place is mentioned during a news cast, the place maybe associated with a related documentary. A notification may bedisplayed to the user that the documentary is available. The user maychoose to override the video element of the news cast with the videoelement of the documentary while the audio element of the news castcontinues to be presented. Alternatively, the user may choose to pausethe news cast in order to view the documentary, or the user may wish toview the documentary and designate the documentary as the primarycontent item. If a user designates a related content item as a primarycontent item, then content that is related to the new primary contentitem—the documentary in this case—can be obtained rather than contentthat is related to the previous primary content item—the news cast inthis case. The previous primary content item may then be a relatedcontent item for the new primary content item. Rules may be applied tothe new primary content item and the newly-designated related contentitem, and those rules may result in a different playback configurationthan rules applied to the content items under their previousdesignations.

Although aspects of the embodiments described in the disclosure willfocus, for the purpose of illustration, on a media device obtainingrelated content from network-accessible servers, one skilled in the artwill appreciate that the techniques disclosed herein may be applied toany number of processes or applications. For example, related contentmay reside on the media device itself, or it may be obtained fromsources other than network-accessible servers, such as via satellite orcable television systems and the like. Various aspects of the disclosurewill now be described with regard to certain examples and embodiments,which are intended to illustrate but not limit the disclosure.

With reference to an illustrative example, a user may wish to consume acontent item and also access separate but related content itemssimultaneously. The user may initiate presentation of a primary contentitem, such as an audio book, on a media device, such as a smart phone orportable media player. The audio book may be associated withsupplemental or related content that may be presented duringpresentation of the audio book. For example, when the user is listeningto an audio book about a songwriter's life, the audio book may containembedded references to related content, such as network addresses atwhich the media device can stream excerpts of songs written by thesongwriter. In such cases, the excerpts of the songs can be mixed withthe audio book (e.g., played simultaneously) at a relevant playback timeor location within the audio book, such as during portions describingthe conception or recording of the songs.

Rules for how content of one type may override other content types maybe defined, either by default or by the user. In the present example,rules may specify that recordings of songs written by the songwriter maybe mixed with the audio book, while other related audio (e.g.,interviews with the songwriter) may be presented while the audio book ispaused. As another example, a video clip may relate to the audio book.If the media device includes video presentation capabilities, a rule mayspecify that the video clip may be presented with its audio muted suchthat the audio book may continue to be presented during playback of thevideo clip.

Content may include both video and audio elements. The audio and videoelements, collectively referred to herein as media elements or simplyelements, may include elements such as audio tracks, video tracks, orother sensory elements of a movie, audio book, song, video game, or someother electronic content. As one example, a content item may be a sportsbroadcast that includes both audio elements (e.g., spoken commentaryregarding the sporting event) and video elements (e.g., video clips orfeeds of the sporting event).

While the sports broadcast is presented, or when presentation isinitiated, references to related content, such as a local radiobroadcast of the sporting event, may be obtained from a separate contentmanagement system or service. Rules can be implemented which allow foror require replacement of the sports broadcast's audio element with thelocal radio broadcast in the case of a live sports broadcast; otherwise,the rules may allow the media device to offer to pause the videoprogramming to listen to the related audio.

In some cases, the initial content that the user wishes to consume, alsoreferred to as the primary content, may include a video element or othervideo portion and the related content may also include a video elementor other video portion. For example, when the user is watching atelevision program and there is a documentary related to a portion ofthe television program, rules may be applied in which the media devicereplaces the video element of what is currently being watched with thevideo element of the related content, but retain the audio element ofthe primary content whenever the primary content is news or“informational” programming. When the primary content is not news, rulesmay be applied in which video elements of both content items arepresented simultaneously in a picture-in-picture (“PIP”) or split-screenformat. If multiple displays are available, the two video elements maybe presented substantially simultaneously on different displays. Forexample, the media device may control two or more video monitors, or themedia device can display one video element on a video monitor andtransmit the second video element, or information about the second videoelement, to a different device such as a tablet computer.

In some embodiments, a user may configure rules for individual contenttypes or genres. For example, the user may configure rules directed totalk shows, news shows, live baseball games, and a specific televisionseries such that any related video content may automatically overridethe video of the primary content (e.g., the video element of the relatedcontent is presented without any prompt to the user), but the relatedaudio content is not presented (e.g., the audio element from the primarycontent item remains the only audio element that is presented). The usermay configure separate rules for audio books, such that when the relatedcontent is music, the music is to be mixed or otherwise simultaneouslypresented with the audio book content, while related audio content thatis not musical is suppressed and the audio book continues to be the onlycontent item that is presented audibly. Alternatively, the user mayconfigure rules such that the audio book is paused and the related audiocontent is automatically played or the user is prompted as to whetherthe related audio content is to be played.

In some embodiments, a supplemental or related content item may beassociated with multiple portions of a primary content item. Forexample, a music clip may be associated with a position within a chapterof an audio book, and also associated with the end of the same chapter.In this case, a user may have multiple opportunities to hear the relateditem. In embodiments where associations of related content would causethe related item to be presented or made available multiple times duringplayback of the primary content, rules may be applied such thatpresentation of the related content or an indication of the availabilityof that related content may be suppressed after the first presentationof the related content or after the first indication that the relatedcontent is available. Under other rules, presentation or indication ofrelated content may be allowed even after the related content has beenfully played.

Related content may be accessed from a network-accessible contentserver. Alternatively, or in addition, related content may be stored ona media device or local storage component associated with a mediadevice. Related content may be retrieved at substantially the same timeas presentation of a primary content item is initiated. Alternatively, amedia device may be configured to retrieve related content periodically.For example, a network enabled television may query a remote contentserver associated with related content every n seconds or minutes duringpresentation of the primary content item in order to determine whethernew related content is available, particularly in the case of live orsubstantially live broadcasts. In still other embodiments, a contentserver or a content management system may be configured to notify amedia device of available supplemental information.

In some embodiments, key portions of the primary content item may bedesignated such that the user is presented with those key portionsregardless of any available related content. For example, crucial plotdevelopments in a movie or audio book may be presented to the userregardless of any available related content item or rule that may beapplicable at the time. Otherwise, the user may lose track of thedevelopment of the primary content item and the user's continuedenjoyment of the primary content item may be jeopardized.

In some embodiments, a user may designate a related content item as aprimary content item, thereby replacing the previous primary contentitem. For example, if the user is watching a broadcast of a sportingevent and the score is such that the game is uninteresting, the user maywish to make a currently presented related content item (e.g., adocumentary, currently being displayed in the secondary presentationwindow of a PIP display, regarding one of the teams) as the primarycontent item. In such cases, content that is related to the new primarycontent item—the documentary in this case—will be obtained, rather thancontent that related to the previous primary content item—the sportingevent in this case.

Networked Content Consumption Environment

Prior to describing embodiments of the related content presentation andrule application process in detail, example networked contentconsumption environments in which the process may be implemented will bedescribed. FIGS. 1A and 1B show example networked content consumptionenvironments and illustrative data flows and interactions betweenvarious entities of the networked content consumption environments.

FIG. 1A illustrates a networked content consumption environment with amedia device 102, a content management system 104, and a content server106. The content management system 104 may store data regarding whichcontent items are related to a primary content item that may bepresented on the media device 104. For example, the content managementsystem 104 can obtain content relationship information from a contentserver 106 at (1) regarding content offered by the content server 106.Other techniques for determining and maintaining content relationshipand reference information are described below.

At (2), a user may use a media device 102 to present a content item,such as an audio book. In this case, the audio book is considered theprimary content item, because it is the content item for which the userinitiated presentation. The media device 102 may be associated with acontent management system 104 for obtaining references to relatedcontent items, or the primary content item itself may be associated withthe content management system 104. In either case, the media device 102can retrieve related content references from the content managementsystem 104 at (3). The related content references may be obtained inbulk (e.g., all content references obtained in one transmission at orsubstantially close to initiation of primary content presentation), orthey may be obtained during presentation of the primary content item.

At (4), the media device 102 may, during playback of the audio book,encounter a portion of the audio book that is associated with areference to related content. In this example, the related content itemmay be a music clip hosted by the content server 106. The media devicecan therefore retrieve or stream the related content item from thecontent server 106 at (5). In some embodiments, the related content itemmay be hosted by the content management system 104 or a differentcontent server, or it may be stored on the media device 102.

At (6), the media device 102 can apply presentation rules to the primarycontent item and the related content item (e.g., the audio book and themusic clip) to determine how the related content item should bepresented. The media device 102 may then present the related contentitem according to the rules. In the present example, the user may haveconfigured a rule that music clips related to audio books may be mixedor overlaid with the audio of the audio book at the proper point withinthe audio book. However, if the related content was a different type ofcontent (e.g., a related audio book), different rules may apply. Forexample, the applicable rules may specify that the user is to beprompted regarding presentation of the related content rather thanautomatically presenting both items simultaneously.

In some embodiments, related content may be hosted by and/or retrievedfrom multiple content servers, rather than from a single content server106. The content management system 104 can determine contentrelationships between various content items hosted by multiple contentservers, and the media device can retrieve related content from theappropriate content server 106.

Turning now to FIG. 1B, another embodiment of a networked contentconsumption environment is illustrated. In the networked contentconsumption environment of FIG. 1B, no separate content managementsystem 104 is used. Rather, the media device may obtain references torelated content directly from the primary content. For example, theprimary content item may include embedded references to related content,or the primary content may be associated with separate data regardingrelated content. In either case, the media device 102 can use thereferences to obtain the related content from content servers 106 a and106 b, among other content sources.

At (1), the media device 102 can present a primary content item withembedded references to related content items. Returning to the exampleabove, the media device may present an audio book. In some embodiments,the primary content may be associated with related content informationthat is not embedded into primary content item. The primary content itemmay be accessed from a storage device associated with the media device102, from removable media, from a network-accessible content server,etc.

At (2), the media device 102 may encounter a portion of the primarycontent item that is associated with a related content item (e.g., areference to a related content item, such as a music clip, is associatedwith the portion of the audio book). At (3), the media device 102 mayretrieve the related content item from a content server 106 a. The mediadevice 102 may apply presentation rules and present the related contentitem, as described above and in detail below (e.g., the media device 102may mix or overlay the audio of the audio book and the music clip forsimultaneous presentation).

Subsequently, the media device may encounter a reference to a secondrelated content item at (5). For example, the media device may encountera reference to video footage of the subject of the audio book. Thereference may indicate that the second related content item is hosted bya different content server 106 b. In some embodiments, multiple contentitems may be hosted by and/or retrieved from a single content server asdescribed above with respect to FIG. 1A, such as the content server 106a that hosted the first related content item.

At (6), the media device 102 can retrieve the second related contentitem from the content server 106 b. The media device 102 can applypresentation rules regarding the second related content item at (7), andpresent some portion of or the entire second related content item asappropriate. For example, a rule may be applied such that because theprimary content item is an audio book, video content may be presentedwith the audio element of the video content suppressed so that the audiomay continue to be presented.

FIG. 2 illustrates a networked content consumption environment includinga communication network 110, a media device 102, a content managementsystem 104, and any number of content servers 106 a-106 n. Thecommunication network 110 may be any wired network, wireless network, orcombination thereof. In addition, the network 110 may be a personal areanetwork, local area network, wide area network, cable network, satellitenetwork, cellular telephone network, or combination thereof. Forexample, the communication network 110 may be a publicly accessiblenetwork of linked networks, possibly operated by various distinctparties, such as the Internet. In some embodiments, the communicationnetwork 110 may be a private or semi-private network, such as acorporate or university intranet. The communication network 110 mayinclude one or more wireless networks, such as a Global System forMobile Communications (GSM) network, a Code Division Multiple Access(CDMA) network, a Long Term Evolution (LTE) network, or some other typeof wireless network. Protocols and components for communicating via theInternet or any of the other aforementioned types of communicationnetworks are well known to those skilled in the art of computercommunications and thus, need not be described in more detail herein.

The media device 102 can correspond to a wide variety of electronicdevices. In some embodiments, the media device 102 can include audio orvisual output components, such as speakers or video screens. Forexample, the media device 102 may be a television, stereo, digital videorecorder (DVRs), digital media receiver, set-top box, desktop computer,server computer or the like. In some embodiments, a media device 102 mayalso be a personal device, such as a smart phone, mobile media player,laptop computer or tablet computer that is optionally configured with avideo display and/or an audio output component, or that may be connectedto external displays, speakers, stereos, or the like. Some media devices102 may include one or more processors and a storage or memory which maycontain software applications executed by the processors. The softwareof the media device 102 may include modules or applications forestablishing communications over the network 110. In addition, thesoftware applications may include multimedia applications which play orotherwise execute audio programs such as music or audio books, videoprograms such as movies or television shows, and video games. Thestorage of the media devices 102 may also contain copies of content toplay on the speakers or video screens.

As further shown in FIG. 2, the media device 102 can include a contentpresentation module 120. The content presentation module 120 can managepresentation of a primary content item to a user, obtaining referencesto related content (e.g., embedded into the primary content item orreceived from the content management system 104), and applying rules forpresenting multiple content items as described in detail below. In someembodiments, the media device 102 may also include a local contentlibrary 122 in which the media device 102 stores primary or relatedcontent items. In some cases, the primary content may be streamed from aseparate network-accessible system or accessed from removable media(e.g., DVDs).

The content management system 104 illustrated in FIG. 2 may correspondto a logical association of one or more computing devices configured tomaintain information regarding related content items and distribute theinformation to media devices 102. The information may reference contentitems hosted by content servers 106, the content management system 104itself, or content that may be stored on a media device 102. Forexample, the content management system 104 may communicate with themedia device 102 via the communication network 110. The media device 102may request data regarding content items related to a currentlypresented or soon-to-be presented content item. The content managementsystem 104 can transmit the requested data to the media device 102. Insome cases, the content management system 104 can transmit the relatedcontent to the media device 102. The content management system 104 canobtain data regarding the related content items from content servers106, from developers of content items, or from an automated analysis ofcontent, such as crowd-source data, social network data, search results,and the like.

The content management system 104 can include a content processingmodule 140, a content library 142, and a related content data store 144.The content processing module 140 can process potential primary contentitems and determine whether supplemental or related content is availablefor portions of the primary content or for the primary content as awhole. For example, the content processing module can access contentsearches, data from content developers and providers, transcripts ofcontent presentations, etc. The content management system 104 can storedata regarding the relationships (e.g., an identifier of the primarycontent, an identifier of the related content, a network address of therelated content, and data regarding the portion of the primary contentto which the related content applies) in the related content data store144. In some cases, the content management system 104 may store primaryand/or related content in a content library 142. For example, a contentdeveloper may establish a content management system 104 that hostsprimary content items in the content library 142 and includespredetermined references to related content in the related content datastore 144.

The content servers 106 a-106 n can correspond to logical associationsof one or more computing devices for hosting content and servicingrequests for the hosted content over the network 110. For example, acontent server 106 a can include a web server component corresponding toone or more server computing devices for obtaining and processingrequests for content (such as streaming video or audio) from the mediadevice 102, the content management system 104, or other devices orservice providers. In some embodiments, one or more content servers 106may be associated with a CDN service provider, an application serviceprovider, etc.

In some embodiments, the networked content consumption environment mayinclude additional or fewer components that those illustrated in FIG. 2.For example, as illustrated in FIG. 1A, a media device 102 may connectdirectly to content servers 102 and not utilize the services of acontent management system 104.

Process for Rule-Based Presentation of Multiple Content Items

FIG. 3 illustrates a process 300 that may be used to for rule-basedpresentation of multiple content items. Advantageously, the process 300may be automatically performed by a media device 102 during presentationof a primary content item such that related content items may bepresented to enhance the user's content consumption experience. Elementsof related content may override elements of the primary contentaccording to certain rules, while elements of the primary content mayoverride elements of the related content according to the same rules orother rules. In addition, users may be interactively prompted todetermine whether to present related content and which portions thereofto present or override.

The process 300 begins at block 302. For example, if the process 300 isimplemented by a media device 102, the process 300 may beginautomatically upon power up or it may be manually initiated by a userwishing to consume content. The process 300 may be embodied in a set ofexecutable program instructions stored on a computer-readable medium,such as one or more disk drives, of a computing system with which themedia device 102 is associated. When the process 300 is initiated, theexecutable program instructions can be loaded into memory, such as RAM,and executed by one or more processors of the computing system. In someembodiments, the computing system may include multiple computingdevices, such as servers, and the processes may be executed by multipleservers, serially or in parallel.

At block 304, the primary content item may be presented. As describedabove, the primary content may be any type of electronic content. Theprimary content may be streamed from a network-accessible server, loadedfrom a local storage device or removable media, etc. The primary contentitem may be associated with various related content items, and dataregarding the related content items may be obtained as described above.

FIG. 4A shows a timeline of an illustrative content item, in this case atelevision broadcast 410 of a sporting event between Team A and Team B.At various points in time (T1, T2 and T3) the television broadcast 410is associated with related content 420, 430 and/or 440. The relatedcontent items may relate to the primary content item 410 as a whole, orto a particular element of the primary content item 410, such as aperson, place, thing, event, etc. Related content 420 is a documentaryregarding construction of the stadium in which the sporting eventbetween Team A and Team B is taking place, while related content 430 isa local radio broadcast of the sporting event in general. Relatedcontent 440 includes video highlights of a record that is broken duringthe sporting event. Because it is difficult to predict when a record maybe broken (or when other events occur in a live broadcast), such relatedcontent items may be associated with the primary content item 410 inreal-time or substantially real-time, as described in detail below.

With continuing reference to FIG. 3, at decision block 306 the mediadevice 102 may determine whether there is related content availableassociated with the primary content 410 at the current presentation timeor content related to a soon-to-be presented portion of the primarycontent item 410. In the present example, the media device 102 maydetermine at times T1, T2, and T3 that a related content item isavailable and proceed to decision block 308. If there is no relatedcontent available at the present time (e.g., a time between T2 and T3 inthe example illustrated in FIG. 4A), the process may proceed to decisionblock 318 where the media device 102 determines whether there is primarycontent 410 remaining to be presented.

At decision block 308, the media device 102 may determine whether topresent to the user an option, prompt or some other notificationregarding the related content item. The media device 102 may store oraccess rules regarding which related content items and types toautomatically present to the user. For example, if the related contentitem is a video and the primary content is a video, rules or policiesmay apply such that the user should be prompted rather thanautomatically pausing or overriding the video element of the primarycontent item 410. However, if the media device 102 is capable ofpresenting multiple video elements simultaneously (e.g., PIP orsplit-screen), then no user prompt may be needed.

In some embodiments, the media device 102 may not have information aboutthe specific characteristics of the content item, such as whether it ishas both video and audio elements or only an audio element. Rather, theinformation available to the media device 102 (e.g., embedded into theprimary content item or received separate from the primary content item)may only indicate the location from which to obtain the related contentitem and/or an identifier of the related content item. In such cases,the media device 102 may prompt the user for a command regarding therelated content item or passively notify the user that related contentis available should the user wish to consume it. The related contentitem may be obtained from its source in order to determine its nature(e.g., whether it is video or audio content) in order to provide theuser with additional information, or the nature of the related contentitem may be determinable from the reference itself (e.g., a fileextension included in the reference).

In the present example, at time T1 the media device 102 may determinethat no user notification should be presented regarding related contentitem 420. Such a determination may be based, for example, on a ruleregarding the presentation of related content during a preliminaryportion of a sporting event broadcast. Under different circumstances(e.g., after the sporting event has begun), different rules may applyand may specify that the user is to be notified of the related contentitem 420 rather than automatically presented with the related contentitem 420. In such cases the process 300 may proceed to block 314, wherethe related content item is retrieved, and subsequently to block 316,where presentation rules are applied.

The related content item 420 includes a video element. In addition, theprimary content item 410, while also including a video element, is in apreliminary period (e.g., pregame). In such cases, the rules applied inblock 316 may specify that the video element of both items is to bepresented via PIP or split-screen display. FIG. 5A illustrates such asimultaneous presentation of video content. As illustrated in FIG. 5A,related content 420 is presented in the main portion of the displayarea, while the primary content 410 is presented in the secondary PIPportion of the display area. In addition, the audio element of therelated content 420 is presented, while the audio element of the primarycontent 410 is overridden or suppressed. The presentation rules mayspecify this approach because the primary content 410 is in apreliminary period. Other periods of the primary content 410 (e.g.,post-game, during commercial breaks) may also result in the primarycontent 410 being displayed in the secondary portion of the display areaif a related content item with a video element is available or iscurrently being presented.

As seen in the content timeline of FIG. 4A, the documentary 420continues after the game begins at time T2. During a subsequentiteration of the process 300 at time T2, rules may be applied whichspecify that the display of the primary content 410 (e.g., the broadcastof the game) is to be changed to the main display area at time T2. Thedocumentary 420 may be changed to the secondary PIP display area, or thepresentation of the documentary 420 may be stopped altogether.

As another example, during a subsequent iteration of the process 300 attime T2 the media device 102 may determine at block 306 that a secondrelated content item 430—the local radio broadcast of the sportingevent—is available. The rules applied at block 316 may indicate thatrelated content of this nature (e.g., an audio-only local broadcast ofthe sporting event) overrides the audio element of the primary contentitem 410. As seen in FIG. 5B, the video element of the primary contentitem 410—the television broadcast—may be presented while the audioelement of the primary content item 410 is suppressed. In its place, theaudio element of the local radio broadcast 430 may be presented.

Returning to FIG. 3, during a subsequent iteration of the process 300 attime T3 the media device 102 may determine at block 306 that a thirdrelated content item 440—the highlights of the previous record—isavailable. At 308, preliminary rules may be applied which specify thatin such cases the user is to be notified of the related content ratherthan automatically presented with the related content. For example, arule may specify that when the primary content 410 is a televisionbroadcast of a sporting event and the sporting event is in progress,related video content is not to be automatically presented. Rather, aprompt or other notification is to be presented to the user so that theuser may select to display the content. As seen in FIG. 5C, an overlaynotification 510 is presented so as to be conspicuous to the user butwithout interfering with consumption of the primary content item 410.The user may activate the notification 510, such as by using a remotecontrol, mouse, touch screen interaction or voice command. Thenotification 510 of FIG. 5C is illustrative only. Other types ofnotifications may be presented, such as more intrusive visual promptswith interface controls that must be activated or cancelled, audiblenotifications such as beeps, and the like.

As seen in FIG. 5D, the highlights 440 may be presented to the user in asecondary PIP display area. Audio from the highlights 440 may besuppressed, in a manner similar to the suppression of the audio from theprimary content 410, due to the presentation of the local radiobroadcast 430. In some cases, different rules may apply, such aspresentation of the highlights in the main display area while theprimary content item is moved to the secondary PIP display area. Theaudio from the highlights 440 may also be presented while the audio fromthe primary content item 410 and the location radio broadcast 430 issuppressed. As yet another example, different rules may apply such thatthe video from the primary content item 410 and audio from the localradio broadcast 430 are both paused while the both video and audioelements of the highlights 430 are presented on the media device 102.

Turning now to FIG. 4B, an illustrative timeline 450 of a combinedcontent presentation is depicted. The presentation in FIG. 4B may beproduced using the same processes and systems disclosed above withrespect to FIG. 4A. The primary content item of FIG. 4B is an audionarration 460 of an article from a travel magazine relating to travel inParis, France. As explained previously, aspects of the presentdisclosure may be used with primary content items of many sorts, whichmay comprise media other than audio. When a primary content item doescomprise an audio recording, the audio may or may not comprise narrationof textual content. For example, the audio may correspond to anunscripted radio show, an audio element of a television show, aso-called “podcast,” a live concert, or any other sort of audio content.When audio does comprise narration of textual content, the text may comefrom a variety of sources other than magazines, including books,newspapers, web sites, plays, etc.

With respect to timeline 450, various attributes of audio narration 460may be identified at various points in time (T1, T2, T3, T4 and T5), andrelated content items associated with the identified attributes may beselected. An attribute of primary content item 460 may include a person,place, object, event, activity, time period, or any other thing that isdescribed, portrayed, mentioned, discussed, or referenced in the primarycontent item. An attribute may also relate to a contextual aspect ofprimary content item 460, and may include, e.g., a genre, mood,location, historical setting, language or time of day. The presence orrelevance of an identified attribute in the primary content item 460 maynot be limited to a single point in time. Rather, an identifiedattribute may continue to be relevant for a period of time following thepoint at which it is first identified, and it may become relevantrepeatedly at various points thereafter. For example, an attribute suchas a historical landmark may be mentioned briefly at the beginning of atravel article, discussed in detail for an extended duration in themiddle of the article, and referred back to at the end of the article.In this example, the same attribute is identified at least three timeswithin a single primary content item, and the relevance of the attributecontinues for various periods of time at each point. Thus, any relatedcontent items associated with an identified attribute may be presentedsubstantially within the various periods of time at which the attributeis relevant to the primary content item, but the precise timing may varyaccording to presentation rules.

The attributes of a primary content item may be identified by analyzingthe primary content item itself, e.g., the audio narration in the caseof primary content item 460, or by analyzing a textual representation ofthe primary content item, such as the magazine article that audionarration 460 is based on. The related content items 470-498 associatedwith the identified attributes of primary content item 460 might comefrom a source that is closely related to the primary content item, or itmight come from an unrelated source. For example, the related contentitems depicted in FIG. 4B include textual excerpts 470-474 and images480-484, all of which may have been published in a magazine along withthe text of the magazine article on which audio narration 460 is based.FIG. 4B also depicts images 490, 492, 498, along with map 494 and audioclip 496, all of which may come from sources other than the magazine inwhich the narrated article was published. As mentioned above withrespect to FIG. 4A, related content items may be associated with primarycontent item 460 in real-time or substantially real-time. Furthermore,the number of attributes and related content items discussed withrespect to FIG. 4B is limited for the sake of clarity, but in someembodiments many more attributes and related content items may beidentified, selected, and presented. Any content item, regardless ofwhether it is a primary or related content item, may be part of somelarger content item. For example, a content item may be part of a largertext (e.g., comprising additional words), a larger audio sequence (e.g.,with a greater temporal duration), a larger video sequence (e.g.,comprising additional video frames), a larger image (e.g., depicting amore expansive scene), or a larger collection of content (e.g.,comprising multiple items in different mediums).

Referring now to timeline 450, at time T1 the audio narration 460 beginswith a discussion of transportation options that a traveler might use toreach Paris. Based on this discussion, attributes such as transportationgenerally or a specific airline that flies to Paris may be identified.Related content associated with these attributes may then be selected.Related content items may include the text excerpt “Getting to Paris,”from the magazine article on which audio narration 460 is based, orimage 490, showing an airplane belonging to the specific airline thatwas identified from the narration. At time T2, audio narration 460 mayturn to a discussion of recommended accommodations in the Paris area.Based on this discussion, attributes such as accommodations generally ora specific recommended hotel may be identified, and related content suchas the text “Where to Stay” or an image of a room from the recommendedhotel may be selected. At time T3, discussion may turn to notabletourist attractions in Paris, beginning with the Eiffel Tower. Thisdiscussion may continue through times T4 and T5 with discussion of NotreDame and the Louvre, respectively. The discussion of sightseeing mayinclude details about the attractions themselves, as well as topics likenavigating Paris by walking or public transportation. Accordingly,attributes such as Paris streets and Paris landscapes may be identified,along with the Eiffel Tower, Notre Dame and the Louvre. Related contentitems selected from these attributes may include text 474 and images480-484, all from the magazine article on which audio narration 460 isbased, along with map 494, audio clip 496, and image 498, all from othersources. Image 480 depicts the Eiffel Tower, image 482 depicts NotreDame, and image 484 depicts the Louvre. Map 494 may depict the streetsof Paris, including a highlighted route between each of the attractionsthat correspond to identified attributes, obtained from a publiclyaccessible mapping service. Audio clip 496 may comprise a recording ofthe cathedral bells at Notre Dame, obtained from a library of stockrecordings. Image 498 may comprise a photograph of a Parisian park,tagged with GPS location information indicating that it is in thevicinity of the Louvre, and obtained from a social media web site.

As described previously with respect to FIG. 3 and FIG. 4A, presentationrules may be applied to determine how related content items should bepresented. In some embodiments, presentation rules may define whichportions of which content items are presented at various times. Inaddition, presentation rules may define a size (e.g., one or morespatial dimensions such as height, width, length or depth) or positionof presentation for a visual element of a content item, or a volumelevel for an audio element of a content item. Presentation rules mayalso specify that multiple content items are presented simultaneously,such as by mixing of audio content, split screen presentation of visualcontent, or combining visual elements from a first set of content itemswith audio elements from a second set of content items. In someembodiments, presentation rules may be used to determine transitions oranimations, comprising specified changes in size, position, volumelevel, or other presentation aspects over a specified period of time.

The presentation rules that are applied to a primary content item andrelated content items may be configured by a user, as described belowwith respect to FIGS. 6A and 6B. In addition, some presentation rulesmay be defined by default, while other presentation rules may betailored to specific content items or groups of content items. Tailoredpresentation rules may be delivered with the corresponding contentitems, such as by embedding the rules in the content items, or byincluding references to the rules in the same data package as thecontent items.

FIGS. 6A and 6B show illustrative user interfaces 600 a and 600 b,respectively, that may be used to configure rules for presentation ofprimary and related content items. As seen in FIG. 6A, a user mayspecify general settings that apply to all video and audio content bydefault. For example, if a multimedia content item is primarilyvideo-based (e.g., sports), settings 602, 604 may be configuredregarding when to override the audio and/or video elements of thedevice. If a multimedia content item is primarily audio-based (e.g.,some news casts), or if the content is only audio (e.g., audio books),then different settings 606, 608 may be configured by the user.

FIG. 6B illustrates an example interface 600 b to configuregenre-specific settings. For example, the settings 652, 654 for sportingevents may be configured independently of the default settings 602, 604for video-based content, as set in the interface 600 a of FIG. 6A. Asanother example, audio books may provide an opportunity to utilizesettings 656, 658 that do not apply to general audio-based content. Forexample, music may be mixed with the audio element of an audio book,while other primary content items that are primarily audio-based may notexperience the same effectiveness from mixing or otherwise combiningrelated content audio elements with the primary content audio element.

Terminology

Depending on the embodiment, certain acts, events, or functions of anyof the processes or algorithms described herein can be performed in adifferent sequence, can be added, merged, or left out altogether (e.g.,not all described operations or events are necessary for the practice ofthe algorithm). Moreover, in certain embodiments, operations or eventscan be performed concurrently, e.g., through multi-threaded processing,interrupt processing, or multiple processors or processor cores or onother parallel architectures, rather than sequentially.

The various illustrative logical blocks, modules, routines, andalgorithm steps described in connection with the embodiments disclosedherein can be implemented as electronic hardware, computer software, orcombinations of both. To clearly illustrate this interchangeability ofhardware and software, various illustrative components, blocks, modules,and steps have been described above generally in terms of theirfunctionality. Whether such functionality is implemented as hardware orsoftware depends upon the particular application and design constraintsimposed on the overall system. The described functionality can beimplemented in varying ways for each particular application, but suchimplementation decisions should not be interpreted as causing adeparture from the scope of the disclosure.

The steps of a method, process, routine, or algorithm described inconnection with the embodiments disclosed herein can be embodieddirectly in hardware, in a software module executed by a processor, orin a combination of the two. A software module can reside in RAM memory,flash memory, ROM memory, EPROM memory, EEPROM memory, registers, harddisk, a removable disk, a CD-ROM, or any other form of a non-transitorycomputer-readable storage medium. An exemplary storage medium can becoupled to the processor such that the processor can read informationfrom, and write information to, the storage medium. In the alternative,the storage medium can be integral to the processor. The processor andthe storage medium can reside in an ASIC. The ASIC can reside in a userterminal. In the alternative, the processor and the storage medium canreside as discrete components in a user terminal.

Conditional language used herein, such as, among others, “can,” “could,”“might,” “may,” “e.g.,” and the like, unless specifically statedotherwise, or otherwise understood within the context as used, isgenerally intended to convey that certain embodiments include, whileother embodiments do not include, certain features, elements and/orsteps. Thus, such conditional language is not generally intended toimply that features, elements and/or steps are in any way required forone or more embodiments or that one or more embodiments necessarilyinclude logic for deciding, with or without other input or prompting,whether these features, elements and/or steps are included or are to beperformed in any particular embodiment. The terms “comprising,”“including,” “having,” and the like are synonymous and are usedinclusively, in an open-ended fashion, and do not exclude additionalelements, features, acts, operations, and so forth. Also, the term “or”is used in its inclusive sense (and not in its exclusive sense) so thatwhen used, for example, to connect a list of elements, the term “or”means one, some, or all of the elements in the list.

Conjunctive language such as the phrase “at least one of X, Y and Z,”unless specifically stated otherwise, is to be understood with thecontext as used in general to convey that an item, term, etc. may beeither X, Y, or Z, or a combination thereof. Thus, such conjunctivelanguage is not generally intended to imply that certain embodimentsrequire at least one of X, at least one of Y and at least one of Z toeach be present.

While the above detailed description has shown, described, and pointedout novel features as applied to various embodiments, it can beunderstood that various omissions, substitutions, and changes in theform and details of the devices or algorithms illustrated can be madewithout departing from the spirit of the disclosure. As can berecognized, certain embodiments of the inventions described herein canbe embodied within a form that does not provide all of the features andbenefits set forth herein, as some features can be used or practicedseparately from others. The scope of certain inventions disclosed hereinis indicated by the appended claims rather than by the foregoingdescription. All changes which come within the meaning and range ofequivalency of the claims are to be embraced within their scope.

What is claimed is:
 1. A system for presenting multiple content items,the system comprising: a data store configured to storecomputer-executable instructions; and a computing device incommunication with the data store, wherein the computing devicecomprises a processor, and wherein the computing device when executingthe computer-executable instructions, is configured to: causepresentation of a primary content item comprising a primary audioelement and a primary video element, wherein the primary content item isassociated with data referencing one or more related content items, eachof the one or more related content items related to at least a portionof the primary content item, and wherein the primary content itemincludes a key portion to be presented regardless of whether any relatedcontent items are related to the key portion; obtain, based at leastpartly on the data referencing the one or more related content items, afirst related content item of the one or more related content items, thefirst related content item comprising at least one of a secondary audioelement and a secondary video element; determine an attribute of contentportrayed in a first portion of the primary content item; determine,based at least partly on the attribute and one or more presentationrules regarding simultaneous presentation of multiple content items, apresentation configuration for the primary content item and the firstrelated content item, the presentation configuration specifying which ofthe primary audio element, the primary video element, the secondaryaudio element, and the secondary video element to present at particulartimes during simultaneous presentation of the primary content item andthe first related content item; cause substantially simultaneouspresentation of the primary content item and the first related contentitem according to the presentation configuration; determine thatpresentation of the key portion of the primary content item will occurat a particular time; and at the particular time, cause presentation ofthe key portion of the primary content item instead of causingsimultaneous presentation of the primary content item and the firstrelated content item according to the presentation configuration.
 2. Thesystem of claim 1, wherein the one or more presentation rules specifythat at least one of an audio element or video element of a content itemis to override at least one of an audio element or video element ofanother content item associated with a particular attribute.
 3. Thesystem of claim 1, wherein the attribute of content portrayed in thefirst portion of the primary content item is associated with at leastone of a person, place, object, event, activity, language, mood,location, time period, or historical setting portrayed in the primarycontent item.
 4. The system of claim 1, wherein the first relatedcontent item is related to one of a person, place, thing, or event ofthe primary content item.
 5. The system of claim 1, wherein the firstrelated content item comprises the first audio element, and wherein thepresentation configuration specifies that the primary audio element andthe secondary audio element are to be presented simultaneously.
 6. Thesystem of claim 1, wherein the first related content item comprises thesecondary video element, and wherein the presentation configurationspecifies that the primary video element and the secondary video elementare to be presented simultaneously.
 7. A computer-implemented method forpresenting multiple content items, the computer-implemented methodcomprising: under control of one or more computing devices configuredwith specific computer-executable instructions, obtaining a primarycontent item comprising a first media element, wherein the primarycontent item is associated with one or more related content items, eachof the one or more related content items related to at least a portionof the primary content item, and wherein the primary content itemincludes a key portion to be presented regardless of whether any relatedcontent items are related to the key portion; obtaining a relatedcontent item of the one or more related content items, the relatedcontent item comprising a second media element, wherein a first portionof the related content item is related to a first portion of the primarycontent item, and wherein a second portion of the related content itemis related to a key portion of the primary content item; determining anattribute of content portrayed in the first portion of the primarycontent item; and determining, based at least partly on the attributeand one or more presentation rules regarding presentation of relatedcontent items during presentation of primary content items, apresentation configuration for the first media element and the secondmedia element at a first particular time corresponding to the firstportion of the primary content item and a second particular timecorresponding to the key portion of the primary content item; causingpresentation of at least one of the first media element and the secondmedia element at the first particular time according to the presentationconfiguration; and causing presentation of the first media element atthe second particular time instead of causing presentation of at leastone of the first media element and the second media element at thesecond particular time according to the presentation configuration. 8.The computer-implemented method of claim 7, wherein the first mediaelement comprises an audio element or a video element.
 9. Thecomputer-implemented method of claim 7, wherein the one or morepresentation rules specify that at least one of an audio element orvideo element of a content item is to override at least one of an audioelement or video element of another content item associated with aparticular attribute.
 10. The computer-implemented method of claim 7,wherein the attribute of content portrayed in the first portion of theprimary content item relates to at least one of a person, place, object,event, activity, language, mood, location, time period, or historicalsetting portrayed in the primary content item.
 11. Thecomputer-implemented method of claim 7, wherein the primary content itemis associated with data referencing the one or more related contentitems, the data indicating to which portion of the primary content itemat least one of the one or more related content items is related. 12.The computer-implemented method of claim 7, further comprisinggenerating a notification regarding the related content item, whereinpresentation of the second media element is further based at leastpartly on a user response to the notification.
 13. Thecomputer-implemented method of claim 7, wherein the first media elementand the second media element are presented simultaneously at the firstparticular time.
 14. The computer-implemented method of claim 7, whereinthe second media element is presented instead of the first media elementat the first particular time.
 15. The computer-implemented method ofclaim 7, wherein the first media element is paused during presentationof the second media element at the first particular time.
 16. Thecomputer-implemented method of claim 7, wherein the primary content itemfurther comprises a third media element, and wherein the related contentitem further comprises a fourth media element.
 17. Thecomputer-implemented method of claim 16, further comprising causingpresentation of the fourth media element at the first particular timeaccording to the one or more presentation rules.
 18. Thecomputer-implemented method of claim 16, further comprising suppressingpresentation of the fourth media element at the first particular timeaccording to the one or more presentation rules.
 19. Thecomputer-implemented method of claim 7 further comprising: receiving anindication that a user has designated the related content item as a newprimary content item; obtaining a second related content item contentitem comprising a third media element, wherein the second relatedcontent item is related to a second portion of the new primary contentitem; and determining, based at least partly on the attribute and theone or more presentation rules, a presentation configuration for thethird media element at a particular time corresponding to the secondportion of the new primary content item.
 20. The computer-implementedmethod of claim 7, wherein the primary content item comprises an audioelement, and wherein determining the attribute of content portrayed inthe first portion of the primary content item comprises analyzing atleast one of the audio element or a textual representation of the audioelement.
 21. The computer-implemented method of claim 7, whereindetermining the presentation configuration comprises: determining afirst presentation rule of the one or more presentation rules thatspecifies that certain media elements of primary content itemsassociated with the attribute are to be suppressed during playback ofcertain media elements of secondary content items; and determining tosuppress the first media element during presentation of the second mediaelement.
 22. A non-transitory computer readable medium comprisingexecutable code that, when executed by a processor, causes a computingdevice to perform a process comprising: obtaining a primary content itemcomprising a first media element, wherein the primary content item isassociated with one or more related content items, each of the one ormore related content items related to at least a portion of the primarycontent item; obtaining a first related content item of the one or morerelated content items, the first related content item comprising asecond media element, wherein the first related content item is relatedto a first portion of the primary content item; obtaining a secondrelated content item of the one or more related content items, thesecond related content item comprising a third media element, whereinthe second related content item is related to a key portion of theprimary content item, and wherein the key portion of the primary contentitem is to be presented regardless of whether any related content itemsare related to the key portion; determining an attribute of contentportrayed in the first portion of the primary content item; determining,based at least partly on the attribute and one or more presentationrules regarding presentation of related content items duringpresentation of primary content items, a presentation configuration forthe first media element and the second media element at a firstparticular time corresponding to the first portion of the primarycontent item and at a second particular time corresponding to the keyportion of the primary content item; and causing presentation of atleast one of the first media element and the second media element at thefirst particular time according to the presentation configuration;causing presentation of the first media element at the second particulartime instead of causing presentation of at least one of the first mediaelement and the second media element at the second particular timeaccording to the presentation configuration.
 23. The non-transitorycomputer readable medium of claim 22, wherein the first media elementcomprises an audio element or a video element.
 24. The non-transitorycomputer readable medium of claim 22, wherein the one or morepresentation rules specify that at least one of an audio element orvideo element of a content item is to override at least one of an audioelement or video element of another content item associated with aparticular attribute.
 25. The non-transitory computer readable medium ofclaim 22, wherein the attribute of content portrayed in the firstportion of the primary content item relates to at least one of a person,place, object, event, activity, language, mood, location, time period,or historical setting portrayed in the primary content item.
 26. Thenon-transitory computer readable medium of claim 22, the process furthercomprising generating a notification regarding the related content item,wherein presentation of the second media element is further based atleast partly on a user response to the notification.
 27. Thenon-transitory computer readable medium of claim 22, wherein the firstmedia element and the second media element are presented simultaneouslyat the first particular time.
 28. The non-transitory computer readablemedium of claim 27, wherein both the first media element and the secondmedia element comprise video elements, and wherein the first mediaelement and the second media element are presented on a single displaydevice in at least one of a split screen format and a picture-in-pictureformat at the first particular time.
 29. The non-transitory computerreadable medium of claim 27, wherein both the first media element andthe second media element comprise video elements, and wherein the firstmedia element and the second media element are presented on separatedisplay devices at the first particular time.
 30. The non-transitorycomputer readable medium of claim 22, wherein the second media elementis presented instead of the first media element at the first particulartime.
 31. The non-transitory computer readable medium of claim 22,wherein presentation of the first media element is paused duringpresentation of the second media element at the first particular time.32. The non-transitory computer readable medium of claim 22, whereinobtaining the related content item comprises streaming the relatedcontent item from a network-accessible server.
 33. The non-transitorycomputer readable medium of claim 22, wherein determining thepresentation configuration comprises: determining a first presentationrule of the one or more presentation rules that specifies that certainmedia elements of primary content items associated with the attributeare to be presented substantially simultaneously with certain mediaelements of secondary content items; and determining to present thefirst media element substantially simultaneously with presentation ofthe second media element.